Operation detection device, operation detection method and projector

ABSTRACT

An operation detection device includes: first and second illuminations that irradiate illumination light from different positions onto an operation surface on which a user performs an operation; a camera that captures the operation surface together with an operation part (finger) of the user; a shadow region extraction unit that extracts first and second shadows of the operation part of the user from a captured image obtained by the camera; a contour detection unit that detects contours of each of the first and second shadows extracted; and a touch point detection unit that detects a touch point of the operation part of the user on the operation surface from the distance between the contours.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the Japanese Patent Application No. 2013-048305filed Mar. 11, 2013, which is incorporated herein by reference in itsentirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an operation detection device and anoperation detection method that detect a finger operation of a user.

2. Description of the Related Art

A technology is proposed that captures an image of an operation part(finger) of a user and extracts the shadows thereof to detect a fingertouch operation, without using a special device such as a touch sensor,as user operation input on a projection surface (screen) of aprojection-type video display device (projector).

JP-2008-59283-A discloses an operation detection device including: ameans for causing an imaging means to capture an image of an operator ina state lit by an illumination means; a means for detecting a region ofa specific site of the operator on the basis of image data of theoperator obtained by the imaging means; a means for extracting shadowportions from the detected region of the specific site of the operator;and a means for detecting, from among the extracted shadow portions, aplurality of line segments in which edges form straight lines, detectingpoints where the detected line segments intersect at acute angles, anddetecting these intersecting points as pointing positions in the regionof the specific site of the operator.

Furthermore, JP-2011-180712-A discloses a projection-type video displaydevice including: a projection unit that projects video onto a screen;an imaging unit for capturing an image of a region including at leastthe video projected onto the screen; an actual image detection unit thatdetects an actual image of a predetermined object that moves above thescreen, from the image captured by the imaging unit; a shadow detectionunit that detects a shadow of the predetermined object produced byprojection light from the projection unit, from the image captured bythe imaging unit; a touch determination unit that determines that thepredetermined object is touching the screen if the distance between theactual image of the predetermined object and a corresponding point ofthe shadow is equal to or less than a predetermined threshold value; anda coordinate determination unit that outputs the coordinates of thepredetermined object as a pointing position with respect to the videowhen it has been determined by the touch determination unit that thereis touching.

SUMMARY OF THE INVENTION

In JP-2008-59283-A, a shadow portion is extracted from the image data ofthe operator obtained by the imaging means, and points where the edgesof the shadow intersect at acute angles are detected as pointingpositions. However, when a hand is open and hovering, because otherfingers can be seen overlapping the shadow of a certain finger, it ispredicted that a plurality of points where the edges of the shadowintersect at acute angles will be generated, and there is therefore arisk of a point that is different from the pointing position beingerroneously detected as a pointing position. Consequently, this methodis not suitable for simultaneously detecting the pointing positions of aplurality of fingers when a hand is open, what is otherwise known as thedetection of a multi-touch operation.

Furthermore, in JP-2011-180712-A, it is determined that a predeterminedobject (finger) is touching a screen if the distance between an actualimage of the predetermined object and a corresponding point of theshadow is equal to or less than a predetermined threshold value.However, when a hand is open, it becomes no longer possible for some ofthe shadows of other fingers to be seen due to the actual image of acertain finger, and it is therefore difficult to detect the actual imageof the finger and a corresponding point of a shadow. In addition,because the distance between the actual image of a finger and the shadowincreases when the hand is open and hovering, for example, the tip endsections of the shadows of other fingers approach the tip end section ofthe actual image of a certain finger, and there is a risk that it mayappear as if the distance between the actual image of the certain fingerand a corresponding point of a shadow has become equal to or less thanthe threshold value, and the finger will be erroneously determined astouching the screen. Consequently, this method is also not suitable forthe detection of a multi-touch operation.

The present invention takes the aforementioned problems intoconsideration, and an object thereof is to provide an operationdetection device and an operation detection method that correctly detecteach of the touch positions of a plurality of fingers even when a handis open, and handle multi-touch operations.

A configuration described in the claims for example is adopted in orderto solve the aforementioned problems.

The present application includes a plurality of units in order to solvethe aforementioned problems, and, to give one example thereof, anoperation detection device of the present invention includes: first andsecond illuminations that irradiate illumination light from differentpositions onto an operation surface; a camera that captures, togetherwith an operation part of a user, the operation surface onto which theillumination light has been irradiated; a shadow region extraction unitthat extracts first and second shadows of the operation part of the userfrom a captured image obtained by the camera; a contour detection unitthat detects contours of each of the first and second shadows extracted;and a touch point detection unit that detects a touch point of theoperation part of the user on the operation surface from the distancebetween the contours. The shadow region extraction unit compares thebrightness of the captured image with a predetermined threshold value,and discerns and extracts the first and second shadows of the operationpart of the user projected by the first and second illuminations, thecontour detection unit extracts, as contours, correspondingsubstantially linear line segments from within the contours of the firstand second shadows, and the touch point detection unit determines thatthe operation part of the user has touched the operation surface whenthe distance between the two extracted contours has become equal to orless than a predetermined threshold value.

According to the present invention, it is possible to correctly detectthe touch positions of a plurality of fingers on an operation surface,and to realize a highly accurate multi-touch operation, withoutproviding a touch sensor or the like on the operation surface.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a configuration diagram of an operation detection deviceaccording to a first embodiment;

FIG. 2A is a front view depicting a user operation in which an operationdetection device is used (a camera is configured internally);

FIG. 2B is a front view depicting a user operation in which an operationdetection device is used (a camera is configured externally);

FIG. 3 is a side view depicting a user operation in which an operationdetection device is used;

FIG. 4A is a drawing depicting the shapes of the shadows of a finger ofa user captured by a camera (in the case of one finger);

FIG. 4B is a drawing depicting the shapes of the shadows of the fingersof a user captured by a camera (in the case of a plurality of fingers);

FIG. 5A is a drawing illustrating a change in the shapes of the shadowsof a finger (top view);

FIG. 5B is a drawing illustrating a change in the contours of theshadows of a finger (as viewed from a camera);

FIG. 6 is a drawing illustrating a method for detecting the contours ofa shadow;

FIG. 7 is a drawing depicting the states of contours when an operationis performed by a plurality of fingers;

FIG. 8 is a drawing depicting the processing flow for touch pointdetection in the first embodiment;

FIG. 9 is a configuration diagram of an operation detection deviceaccording to a second embodiment;

FIG. 10A is a drawing illustrating a change in the shapes of the shadowsof a finger (top view);

FIG. 10B is a drawing illustrating a change in the contours of theshadows of a finger (as viewed from a camera);

FIG. 11 is a drawing depicting the processing flow for touch pointdetection in the second embodiment;

FIG. 12 is a configuration diagram of an operation detection deviceaccording to a third embodiment;

FIG. 13 is a drawing depicting the shapes of the shadows of a fingercaused by a plurality of illuminations;

FIG. 14 is a drawing depicting the processing flow for touch pointdetection in the third embodiment;

FIG. 15 is a configuration diagram of a projector according to a fourthembodiment;

FIG. 16 is a front view depicting the operation state of a shortprojection-type projector;

FIG. 17 is a side view depicting the operation state of a shortprojection-type projector; and

FIG. 18 is an external view depicting an example of a head-mountedprojector.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The embodiments are described hereafter using the drawings.

First Embodiment

In a first embodiment, a description is given with respect to anoperation detection device that uses one camera and two illuminationsarranged in different positions to detect a touch point where anoperation part (finger) of a user touches an operation surface.

FIG. 1 depicts a configuration diagram of an operation detection deviceaccording to the first embodiment. An operation detection device 1includes a camera 100, two illuminations 101 and 102, a shadow regionextraction unit 104, a contour detection unit 105, a touch pointdetection unit 106, a control unit 120, and an output unit 130, andoutputs, to an operation target device 2, detection result data 150 suchas the touch position of a finger on an operation surface. The operationtarget device 2 is a projector for example, receives the detectionresult data 150, and performs video display in accordance with a useroperation. Although all of the elements 100 to 106, 120, and 130 areconfigured inside the operation detection device 1 in FIG. 1, some ofthe constituent elements, the camera 100 and the illuminations 101 and102 for example, may be configured outside the operation detectiondevice 1, and may be connected via a network or a Universal Serial Bus(USB). Although the constituent elements 100 to 106, 120, and 130 areindependent of each other, they may be configured from one or aplurality of constituent elements as required. For example, the elements104 to 106, 120, and 130 may be configured in such a way that theprocessing thereof is performed by one or a plurality of centralprocessing devices (CPUs).

FIG. 2A, FIG. 2B, and FIG. 3 are drawings depicting a state in which auser 3 performs an operation using an operation detection device 1. FIG.2A and FIG. 2B are front views of the operation state, and FIG. 3 is aside view of the operation state. It should be noted that FIG. 2Adepicts the case where the camera 100 and the illuminations 101 and 102have been configured inside the operation detection device 1, and FIG.2B depicts the case where the camera 100 and the illuminations 101 and102 have been configured outside the operation detection device 1. Theuser 3 performs a desired operation by causing a certain finger 30,which constitutes an operation part, to touch a certain position on anoperation surface 22 of a wall surface 21. In the case where theoperation target device 2 is a projector, the operation surface 22 is ascreen that displays projected video, and the user performs an operationon the screen.

The operation detection device 1 is attached to the upper section of thewall surface 21, and the two illuminations 101 and 102 are arrangedoffset in different positions in the horizontal direction on the wallsurface 21, on either side of the camera 100. It should be noted that,in FIG. 2B, the two illuminations 101 and 102 are arranged in the leftand right end sections of the wall surface 21. The finger 30 of the user3 is irradiated by the two illuminations 101 and 102, and the finger 30and the vicinity thereof are captured by the camera 100. The operationdetection device 1 analyses an image captured by the camera 100 anddetects the touch point of the finger from the shapes of the shadows ofthe finger 30 changing when the finger 30 touches the operation surface22.

Next, the operations of the units are described. The camera 100 isconfigured from an image sensor and a lens and so forth, and captures animage including the finger 30 that constitutes the operation part of theuser 3. The two illuminations 101 and 102 are configured fromlight-emitting diodes, circuit boards, and lenses so forth, irradiateillumination light onto the operation surface 22 and the finger 30 ofthe user 3, and project shadows of the finger 30 in the image capturedby the camera 100. It should be noted that the illuminations 101 and 102may be infrared-light illuminations, and the camera 100 may beconfigured from an infrared-light camera. It is thereby possible for aninfrared-light image captured by the camera 100 to be separated from avisible-light image projected by the operation target device 2(projector) and acquired.

The shadow region extraction unit 104 is configured from a circuit boardor software or the like, and extracts shadows from an image obtained bythe camera 100 and generates a shadow image. For example, the backgroundimage of the operation surface 22, which is captured in advance, issubtracted from an image captured during the detection of an operationand a difference image is generated, the brightness of the differenceimage is binarized using a predetermined threshold value Lth, andregions that are equal to or less than the threshold value arepreferably taken as shadow regions. In addition, processing is performedin which shadow regions that are not mutually connected to the extractedshadows are each discerned as separate shadows, what is otherwise knownas labeling processing. As a result of the labeling processing, it ispossible to identify which fingers correspond to the plurality ofextracted shadows.

The contour detection unit 105 is configured from a circuit board orsoftware or the like, and extracts the contours of the shadow regionsfrom the shadow image. For example, contours are obtained by scanningwithin a shadow image in a constant direction (from the top-left to thebottom-right) to determine a starting pixel for contour tracking, andtracking the neighboring pixels of the starting pixel in acounterclockwise manner. A method for detecting contours is describedusing FIG. 6. It should be noted that the processing of the shadowregion extraction unit 104 and the contour detection unit 105 is notrestricted to the aforementioned method, and another image processingalgorithm may be used.

The touch point detection unit 106 is configured from a circuit board orsoftware or the like, and, on the basis of the shapes and positions ofcontours, determines the touch state of the finger 30 with respect tothe operation surface 22, and also detects a touch point (coordinates).A method for detecting a touch point is described using FIGS. 5A and 5Band FIG. 8.

The control unit 120 is configured from a circuit board or software orthe like, and controls the illuminations 101 and 102, the shadow regionextraction unit 104, the contour detection unit 105, the touch pointdetection unit 106, and the output unit 130 on the basis of a capturedoperation in an image captured by the camera 100.

The output unit 130 is an interface that outputs the detection resultdata 150 to the operation target device (projector) 2, which constitutesthe operation target, and is configured from a network connection or aUniversal Serial Bus (USB) connection, an ultrasound unit, or aninfrared-ray communication device or the like. Touch state informationregarding whether or not the finger 30 is touching the operation surface22 and touch point coordinates are included in the detection result data150.

FIG. 4A and FIG. 4B are drawings depicting the shapes of the shadows ofthe fingers of a user captured by a camera. FIG. 4A depicts the case ofone finger, and FIG. 4B depicts the case of a plurality of fingers inwhich a hand is open. The way in which the shapes of the shadows changeis indicated by state (a) in which the finger 30 of the user is nottouching the operation surface 22 (paper surface) and state (b) in whichthe finger 30 of the user has touched the operation surface 22.

In FIG. 4A, in state (a) in which the finger 30 of the user is nottouching the operation surface 22, two shadows 401 and 402 (indicated bydiagonal lines) caused by the two illuminations 101 and 102 at the leftand right of the finger 30 are formed. The shadow 401 on the left sideis projected by the illumination 102 on the right side, and the shadow402 on the right side is projected by the illumination 101 on the leftside. These two shadows 401 and 402 are separate from each other.Meanwhile, in state (b) in which the tip end section (fingertip) of thefinger 30 is touching the operation surface 22, the two shadows 401 and402 have approached the position of a fingertip 30 a. It should be notedthat although partial regions at the tip-end sides of the shadows 401and 402 are hidden behind the finger 30, these hidden portions are notincluded in the shadow regions.

Meanwhile, although FIG. 4B depicts the case of a plurality (five) offingers 31, 32, . . . 35 in which the hand is open, this is basicallythe same as the case of one finger in FIG. 4A. In state (a) in which thefingers are not touching the operation surface, shadows (indicated bydiagonal lines) are formed to the left and right of each of the fingers.However, in this state, because several of the shadows are hidden byother fingers and overlap the shadows of the other fingers, the shadowsthat can be clearly seen are the shadow 411 on the left side of thefinger 31, the shadow 421 on the left side of the finger 32, the shadow442 on the right side of the finger 34, and the shadow 452 on the rightside of the finger 35 and so forth. Meanwhile, in state (b) in which thefingers are touching the screen, the two shadows of each finger havebecome clear and have approached the positions of the fingertips. Here,reference numerals are appended and indicated with respect to the twoshadows 411 and 412 of the finger 31, and the two shadows 421 and 422 ofthe finger 32.

FIG. 5A and FIG. 5B are drawings illustrating a method for determining atouch from the shapes of the shadows of a finger. FIG. 5A is a drawingillustrating a change in the shapes of shadows as viewed from above (theceiling side) the operation surface 22. FIG. 5B is a drawingillustrating a change in the contours of shadows when viewing theoperation surface 22 from the camera side. State (a) in which the finger30 of the user is not touching the operation surface 22 and state (b) inwhich the finger 30 of the user has touched the operation surface 22 arecompared in both drawings. Although a description is given here withrespect to the case of one finger 30 (FIG. 4A) for simplicity, thedescription also applies to the case of a plurality of fingers (FIG.4B).

In FIG. 5A, in state (a) in which the finger 30 is not touching theoperation surface 22, light from the two illuminations 101 and 102 isblocked by the finger 30, and the shadows 402 and 401 (indicated bydiagonal lines) are formed. At this time, because an actual image of thefinger 30 is captured at the front in the image captured by the camera100, the portion at the rear side of the finger 30 is not captured as ashadow. As a result, in the image captured by the camera 100, the twoshadows 401 and 402 are separate from each other. Here, the distancebetween the outer sides of the shadows 401 and 402 is taken as d. Thisdistance d is dependent upon the space s between the finger 30 and theoperation surface 22, and the distance d increases as the space sincreases. Meanwhile, in state (b) in which the finger 30 is touchingthe operation surface 22, because the space s=0, the distance betweenthe outer sides of the shadows 401 and 402 is the smallest value d0.Therefore, by measuring the distance d between the outer sides of theshadows 401 and 402, it is possible to determine whether or not thefinger 30 has touched the operation surface 22.

In FIG. 5B, shadows in which the shape of the finger 30 is projected arecaptured in the camera image. A contour 501 on the left side (outerside) of the shadow 401 formed on the left side of the finger 30, and acontour 502 on the right side (outer side) of the shadow 402 formed onthe right side are then detected from within the contours of theshadows. With regard to these contours, only corresponding substantiallylinear line segments are extracted from within the contours thatsurround the shadows, and the curved portions of the fingertips are notincluded. The shortest distance d between the two extracted contours 501and 502 is then obtained as the distance between the two shadows 401 and402. In this case, the contours 501 and 502 are not parallel, andordinarily the shortest distance d is determined between the endsections 501 a and 502 a at the fingertip side.

In state (a) in which the finger 30 is not touching the operationsurface 22, because the two shadows 401 and 402 are separate from eachother, the shortest distance d between the two contours 501 and 502 alsobecomes a large value. Meanwhile, in state (b) in which the finger 30 istouching the operation surface 22, the two shadows 401 and 402 haveapproached each other, and the two contours 501 and 502 have alsoapproached each other. The shortest distance therebetween becomes thesmallest value d0 at the end sections 501 a and 502 a at the fingertipside. Consequently, by defining a predetermined threshold value dth(here, dth>d0), and determining whether or not the shortest distance dbetween the contours 501 and 502 is within the threshold value dth, itis possible to determine whether or not the finger 30 is touching theoperation surface 22. In this example, because the contours 501 and 502are extracted from the outer side portions of the two shadows 401 and402, the change in the distance between the contours is notable comparedto when extraction is performed from the inner side portions, and as aresult there is an improvement in touch detection accuracy.

If it has been determined that the finger 30 has touched the operationsurface 22, a center point P of the end sections 501 a and 502 a of thecontours 501 and 502 is determined as the touch point, and thecoordinate values of the touch point on the operation surface 22 arecalculated. It should be noted that, in order to accurately calculatethe coordinates of the touch point, a correction to offset thecoordinates of the center point P in the fingertip direction by apredetermined amount may be implemented as required.

FIG. 6 is a drawing illustrating a method for detecting the contours ofa shadow. As depicted in FIG. 6(a), an image 40 of the shadow of a handis included within an imaging screen 90. When the left-side contours areto be obtained, the presence/absence of shadow pixels is detected inscanning lines 91 (solid lines) going from the left to the right sidewithin the screen 90. At this time, a position where the scanned pixelswitches from a non-shadow pixel (taken as pixel 0) to a shadow pixel(taken as pixel 1) becomes a left-side contour. When a plurality offinger shadows are present as in the drawing, the pixels switch frompixel 0 to pixel 1 at each finger shadow, and a plurality of contoursare detected. The left-side contour 50L depicted in (b) is obtained inthis way.

Likewise, when the right-side contours are to be obtained, thepresence/absence of shadow pixels is detected in scanning lines 92(dotted lines) going from the right to the left side within the screen90. The right-side contour 50R depicted in (c) is obtained in this way.Curved portions such as fingertips are removed from among the contours50L and 50R obtained in this way, and contours made up of substantiallylinear line segments are detected. It should be noted that theaforementioned method is an example, and another algorithm may be usedfor the detection of contours.

FIG. 7 is a drawing depicting the states of contours when an operationis performed by a plurality of fingers. As in (a), when the plurality offingers 31, 32 . . . have been made to touch the operation surface whilethe hand 3 a is open, the left-side shadows 411, 421 . . . and theright-side shadows 412, 422 . . . are formed with respect to thefingers. (b) depicts the contours thereof, and depicts the left-sidecontours 511L, 521L . . . of the left-side shadows 411, 421 . . . , andthe right-side contours 512R, 522R . . . of the right-side shadows 412,422 . . . . The shortest distance between the corresponding contours ofthe fingers becomes the smallest value d0 in the vicinity of thefingertips thereof. This distance becomes substantially equal even ifthe fingertip direction is inclined from the vertical direction as withthe finger 31. Thus, according to the present embodiment, the touches ofa plurality of fingers can be independently detected even when the handis open, and it becomes possible to apply the present embodiment to amulti-touch operation.

FIG. 8 is a drawing depicting the processing flow for touch pointdetection in the first embodiment.

In S1000, the operation detection device 1 starts processing fordetecting the touch point of a finger. Illumination light is irradiatedfrom the two illuminations 101 and 102 due to an instruction from thecontrol unit 120, and the operation surface is captured by the camera100.

In S1001, the shadow region extraction unit 104 subtracts the backgroundfrom the image captured by the camera 100 and obtains a differenceimage, and portions where the brightness is equal to or less than thethreshold value Lth are extracted as shadow regions. In S1002, theshadow region extraction unit 104 performs processing in which shadowregions that are not mutually connected to the extracted shadows areeach discerned as separate shadows, what is otherwise known as labelingprocessing.

In S1003, the contour detection unit 105 detects contours with respectto the shadows that have been subjected to labeling processing. Forexample, as in FIG. 5B, the contour 501 on the left side of the shadow401, and the contour 502 on the right side of the shadow 402 aredetected. Here, when there are a plurality of finger shadows, thedetermining of a pair of shadows corresponding to a specific finger isperformed on the basis of labeling processing, and the left/rightswitching between contours with respect to the shadows is performed onthe basis of the procedure in FIG. 6.

In S1004, the touch point detection unit 106 determines whether or notthere is a place where the shortest distance d between the detectedcontours 501 and 502 is equal to or less than the threshold value dth.This threshold value dth is defined in such a way that it is possible toidentify the distance d0 between the end sections 501 a and 502 a of thecontours when a finger is touching as in FIG. 5B. If the result of thedetermination is that there is such a place, processing advances toS1005. If there is no such a place, processing returns to S1001, and theaforementioned processing is repeated.

In S1005, the touch point detection unit 106 determines, as a touchpoint, the center point P of the place (place where the shortestdistance d is equal to or less than the threshold value dth) detected inS1004, and calculates the coordinate values of the touch point on theoperation surface 22. The output unit 130 outputs the calculated touchpoint coordinates as detection result data 150.

In S1006, it is determined whether the touch point detection is to becontinued due to an instruction or the like from the user, and if thetouch point detection is to be continued, processing returns to S1001and the aforementioned processing is repeated.

As described above, the operation detection device of the firstembodiment uses one camera and two illuminations to detect the contoursof two shadows projected by the two illuminations. A touch point is thendetected from a place where the shortest distance between the contourshas approached within a predetermined distance, and the coordinates ofthe touch point are output. In this method, because the touches of aplurality of fingers can each be independently detected when the hand isopen, it is possible for detection to be performed correctly even withrespect to a multi-touch operation.

The operation target device 2 has been described using the example of aprojector; however, it is also possible for the operation target device2 to be applied to a general display or a head-mounted display or thelike. The operation surface is not restricted to a screen, and can beapplied to any type of surface such as a wall surface or a table.

Second Embodiment

In a second embodiment, a configuration is implemented in which thetouch point of a finger is detected by alternately turning on the twoilluminations 101 and 102 of the operation detection device 1.

FIG. 9 depicts a configuration diagram of an operation detection deviceaccording to the second embodiment. The difference with the firstembodiment (FIG. 1) is that a switch 110 for illumination switching hasbeen added. The switch 110 is configured from a circuit board and soforth, and alternately turns on the illumination 101 and theillumination 102 in accordance with an instruction from the control unit120. At such time, the control unit 120 switches the imaging performedby the camera 100, in accordance with the timing at which theillumination 101 and the illumination 102 are turned on. Therefore, onlythe shadows projected by the illumination 101 are captured in an imageof a certain timing (frame 1) captured by the camera 100 for example,and only the shadows projected by the illumination 102 are captured inan image of the next timing (frame 2). Consequently, images are obtainedin which the positions of finger shadows switch in each frame, and it ispossible for the touch of a finger to be easily detected from thedistance between the two shadows.

The second embodiment has a feature in that, because shadows areextracted by alternately turning on the illuminations, two shadows canbe temporally separated and detected even if the two shadows partiallyoverlap. Therefore, it is possible for a touch point to be correctlydetected even if the two illuminations 101 and 102 are installed on thesame side of the camera 100, and two shadows are formed on the same sideof a finger and partially overlap. In the following example, adescription is given with respect to the case where the illuminations101 and 102 are installed on the same side of the camera 100. Naturally,it goes without saying that the second embodiment is effective also whenthe two illuminations 101 and 102 are installed on mutually oppositesides on either side of the camera 100.

FIG. 10A and FIG. 10B are drawings depicting changes in the shapes ofthe shadows of a finger when the two illuminations 101 and 102 areinstalled on the same side of the camera 100. FIG. 10A is a top view asviewed from above (the ceiling side) the operation surface 22. FIG. 10Bis a drawing in which the operation surface 22 is viewed from the cameraside. State (a) in which the finger 30 of the user is not touching theoperation surface 22 and state (b) in which the finger 30 of the userhas touched the operation surface 22 are compared in both drawings.

When the illumination 101 is turned on in frame 1, a shadow 401 of thefinger 30 is formed, and when the illumination 102 is turned on in frame2, the shadow 402 is formed. Both of the shadows 401 and 402 are on thesame side (the right side in the drawings) of the finger 30.

The contours of the shadows in this case are, as depicted in FIG. 10B,both extracted from the outer sides of the shadows as viewed from thefinger 30 (the right side in the drawings). In other words, the contour501 of the shadow 401 is extracted in frame 1, the contour 502 of theshadow 402 is extracted in frame 2, and the shortest distance d′ betweenthe contours 501 and 502 is obtained. Where the distance d′ becomes theshortest is ordinarily near the fingertips at the end sections of thecontours (indicated by the white circle marks). The distance d′ isdependent upon the space s between the finger 30 and the operationsurface 22, and becomes the smallest value d0′ when the finger touchesthe operation surface. It should be noted that this value d0′ isdifferent from the value d0 in the first embodiment (FIG. 5A, FIG. 5B),and becomes a smaller value. In this case, by defining a predeterminedthreshold value dth′ (here, dth′>d0′), and determining whether or notthe shortest distance d′ between the contours 501 and 502 is within thethreshold value dth′, it is possible to determine whether or not thefinger 30 is touching the operation surface 22.

FIG. 11 is a drawing depicting the processing flow for touch pointdetection in the second embodiment. Here, a description is given withrespect to the case where, as depicted in FIG. 10A and FIG. 10B, the twoilluminations 101 and 102 are installed on the same side of the camera100, and a touch point is detected from the two shadows 401 and 402produced at the finger 30.

In S1100, the operation detection device 1 starts processing fordetecting the touch point of a finger.

In S1101, at the timing of frame 1, the illumination 101 is turned ondue to an instruction from the control unit 120, and an image iscaptured by the camera 100. The shadow region extraction unit 104extracts, from the captured image, the shadow 401 formed on the rightside of the finger 30. In S1102, the contour detection unit 105 detectsthe contour 501 on the right side of the shadow 401.

In S1103, at the timing of frame 2, the illumination 102 is turned ondue to an instruction from the control unit 120, and an image iscaptured by the camera 100. The shadow region extraction unit 104extracts, from the captured image, the shadow 402 formed on the rightside of the finger 30. In S1104, the contour detection unit 105 detectsthe contour 502 on the right side of the shadow 402.

In S1105, the touch point detection unit 106 determines whether or notthere is a place where the shortest distance d′ between the detectedcontours 501 and 502 is equal to or less than the threshold value dth′.This threshold value dth′ is defined in such a way that it is possibleto identify the distance d0′ between the contours when a finger istouching in FIG. 10B. If the result of the determination is that thereis such a place, processing advances to S1106. If there is no such aplace, processing returns to S1101, and the aforementioned processing isrepeated.

In S1106, the touch point detection unit 106 determines, as a touchpoint, the vicinity (left side) P′ of the place (place where theshortest distance d′ is equal to or less than the threshold value dth′)detected in S1105, and calculates the coordinate values of the touchpoint on the operation surface 22. The output unit 130 outputs thecalculated touch point coordinates as detection result data 150.

In S1107, it is determined whether the touch point detection is to becontinued due to an instruction or the like from the user, and if thetouch point detection is to be continued, processing returns to S1101and the aforementioned processing is repeated.

In the aforementioned processing flow, a description has been given withrespect to the case where the two illuminations 101 and 102 areinstalled on the same side of the camera 100; however, when the twoilluminations 101 and 102 are installed on opposite sides on either sideof the camera 100, it is preferable for the contour 501 on the left sideof the shadow 401 and the contour 502 on the right side of the shadow402 to be detected as depicted in FIG. 5, and the shortest distancetherebetween to be obtained.

As described above, the operation detection device of the secondembodiment uses one camera and two illuminations and alternately turnson the two illuminations to thereby detect the contours of two shadowsprojected by each of the two illuminations. A place where the shortestdistance between the contours has approached within a predetermineddistance is then determined as a touch point, and the coordinates of thetouch point are output. In the second embodiment, because the twoshadows are able to be temporally separated and extracted, it ispossible for detection to be performed correctly even if the twoilluminations are installed on the same side of the camera. Therefore,the degree of freedom with regard to the installation of theilluminations increases.

Third Embodiment

In a third embodiment, a configuration is implemented in which the touchpoint of a finger is detected by using a plurality of illuminations forthe illuminations of the operation detection device 1.

FIG. 12 depicts a configuration diagram of an operation detection deviceaccording to the third embodiment. The difference with the firstembodiment (FIG. 1) is that the two illuminations 101 and 102 have beenreplaced by a plurality of (N) illuminations 103. The plurality ofilluminations 103 are arranged at mutually offset positions with respectto the camera 100, and are all turned on at the same time.

FIG. 13 is a drawing depicting the shapes of the shadows of a finger ofa user captured by a camera when the plurality of illuminations 103 areused. Here, the case where eight (N=8) illuminations are arranged withfour illuminations on each side of the camera 100 is depicted. The casewhere the finger 30 is not touching the operation surface 22 is depictedin (a), the case where the finger 30 has touched the operation surface22 is depicted in (b), and the method for determining the touch point isdepicted in (c). As depicted in (a), when the finger 30 is not touchingthe operation surface 22, a plurality (N=8) of shadows 401 to 408 thatare projected by the plurality of illuminations 103 on the left andright of the finger 30 are formed. When the finger approaches theoperation surface 22, overlapping sections 401′ to 408′ are producedbetween adjacent shadows.

As depicted in (b), when the finger 30 is touching the operation surface22, the plurality of shadows 401 to 408 concentrate at the fingertip 30a. As a result, a portion in which the plurality of shadows overlap isproduced in the vicinity of the fingertip 30 a. When the shadowsoverlap, the density (darkness) of the shadows increases in accordancewith the number of overlapping shadows, and in the regions 409′, all(four shadows on each side) of the shadows overlap and the shadowdensity becomes the maximum. Maximum shadow density sections 409′ areextracted by defining a brightness threshold value Lth′ based on thefact that the shadow brightness in the maximum shadow density sections409′ is the lowest.

In (c), the maximum shadow density sections 409′ (where the shadowbrightness is equal to or less than the threshold value Lth′) areextracted on both sides of the finger, and a region 410 surroundingthese is defined as a touch region. The highest fingertip-side sectionP″ of the touch region 410 is then determined as the touch point of thefinger.

FIG. 14 is a drawing depicting the processing flow for touch pointdetection in the third embodiment.

In S1200, the operation detection device 1 turns on the plurality ofilluminations 103, and starts processing for detecting the touch pointof a finger.

In S1201, the shadow region extraction unit 104 extracts a plurality ofshadows from an image captured by the camera 100. At such time, in theshadow region extraction unit 104, the brightness threshold value Lth′for when shadows are extracted is defined in such a way that the maximumshadow density sections 409′ in which a plurality of shadows overlap areextracted.

In S1202, in the shadow region extraction unit 104, it is determinedwhether or not it has been possible for the maximum shadow densitysections 409′ to be extracted. If it has been possible for the shadows409′ to be extracted, processing advances to S1203. If extraction is notpossible, processing returns to S1201.

In S1203, the contour detection unit 105 determines the region 410 thatsurrounds the maximum shadow density sections 409′ extracted in S1202.This region 410 becomes the finger touch region.

In S1204, the touch point detection unit 106 determines, as a touchpoint, the highest fingertip-side section P″ of the region 410 extractedin S1203, and calculates the coordinate values of the touch point on theoperation surface 22.

In S1205, it is determined whether the touch point detection is to becontinued due to an instruction or the like from the user, and if thetouch point detection is to be continued, processing returns to S1201and the aforementioned processing is repeated.

As described above, the operation detection device of the thirdembodiment uses one camera and a plurality of illuminations to detect aportion in which the density of shadows projected by the plurality ofilluminations is the maximum. It is possible to determine a touch pointfrom this portion in which the shadow density is the maximum. Thisprocedure can be applied to the case where a plurality of shadowsoverlap, and the number of illuminations may be an arbitrary numberequal to or greater than two. In the present embodiment, because thetouch point can be detected merely by determining the density(brightness) of shadows, there are effects that there are few processingsteps and the detection speed is increased.

Fourth Embodiment

In a fourth embodiment, a description is given with respect to theconfiguration of a projector that has the aforementioned operationdetection device 1 incorporated therein.

FIG. 15 depicts a configuration diagram of a projector 2 a (2 b)according to the fourth embodiment. Here, the projector 2 a (2 b) hasthe operation detection device 1 mentioned in the first embodiment(FIG. 1) incorporated therein, and also has a configuration forprojecting video added thereto as a projector function. The projector 2a (2 b) has, as the functions of a projector, a central processing unit201, an operation analysis unit 202, a memory 203, a video control unit204, and a video projection unit 210.

The central processing unit 201 is configured from a semiconductor chipsuch as a central processing device (CPU) and software such as anoperating system (OS), and on the basis of user operations and so forthdetected by the operation analysis unit 202, controls the input andoutput of information to the memory 203, and each of the units such asthe video control unit 204 and the video projection unit 210.

The operation analysis unit 202 is configured from a circuit board orsoftware or the like, and on the basis of the coordinates of a touchpoint obtained from the output unit 130 of the operation detectiondevice 1, detects a user operation with respect to projected video bydetermining the correspondence between the video being projected and thetouch point.

The memory 203 is configured from a semiconductor and so forth, andstores information required for calculations and control performed bythe central processing unit 201, and video information and so forth thatis displayed as projected video.

The video control unit 204 is configured from a circuit board and soforth, performs calculation processing required for drawing videoinformation, in accordance with control performed by the centralprocessing unit 201, and outputs drawing information made up of a set ofpixels, in a format suitable for input to the video projection unit 210.

The video projection unit 210 is configured from a light source such asa lamp, optical components such as a lens and a reflection mirror, and aliquid crystal panel and so forth, modulates beams of light emitted fromthe light source, forms image light corresponding to the drawinginformation sent from the video control unit 204, and expands andprojects the image light onto a projection surface such as a screen.

It should be noted that, although the units of FIG. 15 are independentof each other, they may be configured from one or a plurality ofconstituent elements as required. For example, units 201 to 204 may beconfigured in such a way that the processing thereof is performed by oneor a plurality of semiconductor chips (system-on-a-chip (SoC) or thelike).

FIG. 16 and FIG. 17 are external views depicting an example of a shortprojection-type projector 2 a as the projector. FIG. 16 is a front viewdepicting a state in which a user is performing an operation, and FIG.17 is a side view. The short projection-type projector 2 a is attachedto the upper section of the wall surface 21. By emitting projectionlight 23 a from the video projection unit 210 on the basis of apredetermined video signal such as a GUI, projected video 23 isprojected onto a screen 22′ on the wall surface 21. The user 3 performsa finger operation on the screen 22′ that also serves as an operationsurface, and is thereby able to control the display and so forth of theprojected video 23.

When the user 3 touches an arbitrary place of the projected video 23with a finger 30, the operation detection device 1 detects a touch pointfrom a shadow image of the finger, and sends the detection data to theoperation analysis unit 202 by way of the central processing unit 201.The operation analysis unit 202 analyses the operation content withrespect to the projected video 23, and the central processing unit 201executes processing such as a video alteration corresponding to the useroperation. By incorporating the operation detection device 1 inside theprojector 2 a in this way, the user is able to efficiently perform anoperation with respect to projected video, and, particularly in thepresent embodiment, is able to suitably perform a multi-touch operation.

FIG. 18 is an external view depicting an example of a head-mountedprojector 2 b as another configuration of the projector. In thehead-mounted projector 2 b, a small projector main body 20 is attachedto a spectacles-type housing, video 23 is projected by projection light23 a being emitted onto the lens surfaces of the spectacles, and theuser is able to view video.

Furthermore, an illumination 101 and an illumination 102 are attached toboth ends of the spectacles-type housing, a camera 100 is attached tothe center of the housing, and together with it being possible toirradiate an operation surface 22 that is in the line of sight of theuser, it is possible to capture a finger operation performed by the useron the operation surface 22 and to detect the touch point thereof.

Therefore, the video 23 projected onto the lens surfaces of thespectacles by the small projector main body 20 and the operation surface22 on which the user performs an operation overlap in the field ofvision of the user and behave as if the video is being displayed on theoperation surface 22. In other words, when the small projector main body20 has displayed a video, the user is able to perform a multi-touchoperation with respect to the displayed video by touching the operationsurface 22 with a fingertip.

As described above, by incorporating the operation detection deviceinside the projector, an effect is obtained in that it is possible forprojected video to be operated in a multi-touch manner without providinga sensor or the like on a video projection surface.

It should be noted that the present embodiments described above areexemplifications for describing the present invention, and are notintended to restrict the scope of the present invention to only theembodiments.

What is claimed is:
 1. An operation detection device that detects an operation performed by a finger of a user with respect to an operation surface, the operation detection device comprising: a first illumination and a second illumination that irradiate illumination light from different positions onto the operation surface; a camera that captures, together with the finger of the user, the operation surface onto which the illumination light has been irradiated; a shadow region extraction unit that, on the basis of a captured image obtained by the camera, extracts a first shadow of the finger of the user produced by the first illumination and a second shadow of the finger of the user produced by the second illumination; a touch point detection unit that, on the basis of the first shadow and the second shadow extracted, detects a touch point of the finger of the user on the operation surface; and a contour detection unit that detects a contour of the first shadow and a contour of the second shadow extracted, wherein the touch point detection unit detects a touch point of the finger of the user on the operation surface from the distance between the contour of the first shadow and the contour of the second shadow, wherein the shadow region extraction unit compares the brightness of the captured image with a predetermined threshold value, and discerns and extracts the first shadow of the finger of the user and the second shadow of the finger of the user, wherein the contour detection unit extracts, as contours, corresponding substantially linear line segments from within the contour of the first shadow and the contour of the second shadow, and wherein the touch point detection unit determines that the finger of the user has touched the operation surface when the distance between the two contours extracted has become equal to or less than a predetermined threshold value.
 2. The operation detection device according to claim 1, wherein the first illumination and the second illumination irradiate in a temporally alternating manner, the camera captures the operation surface in accordance with irradiation timings of each of the irradiation of the first illumination and the irradiation of the second illumination, and the shadow region extraction unit extracts the first shadow from an image captured by the first illumination, and extracts the second shadow from an image captured by the second illumination temporally separated from the image captured by the first illumination.
 3. The operation detection device according to claim 1, wherein the first illumination and the second illumination are installed in such a way that the illumination directions thereof are oriented toward substantially the same side as the imaging direction of the camera.
 4. An operation detection method that detects an operation performed by a finger of a user with respect to an operation surface, comprising the steps of: irradiating illumination light from different positions onto the operation surface by a first illumination and a second illumination; capturing, by a camera, together with the finger of the user, the operation surface onto which the illumination light has been irradiated; extracting a first shadow of the finger of the user produced by the first illumination and a second shadow of the finger of the user produced by the second illumination on the basis of a captured image obtained by the camera; and detecting a touch point of the finger of the user on the operation surface on the basis of the first shadow and the second shadow extracted, wherein corresponding substantially linear line segments are extracted as contours from within contours of the first shadow and the second shadow extracted, and wherein it is determined that the finger of the user has touched the operation surface when the distance between the two contours extracted has become equal to or less than a predetermined threshold value.
 5. The operation detection method according to claim 4, wherein the first illumination and the second illumination are made to irradiate in a temporally alternating manner, the operation surface is captured by the camera in accordance with irradiation timings of each of the irradiation of the first illumination and the irradiation of the second illumination, and the first shadow is extracted from an image captured by the first illumination, and the second shadow is extracted from an image captured by the second illumination temporally separated from the image captured by the first illumination.
 6. A projector comprising: a video projection unit that projects video; a first illumination and a second illumination that irradiate illumination light from different positions onto an operation surface at least part of which overlaps a video surface projected by the video projection unit; a camera that captures/together with the finger of the user, the operation surface onto which the illumination light has been irradiated; a shadow region extraction unit that, on the basis of a captured image obtained by the camera, extracts a first shadow of the finger of the user produced by the first illumination and a second shadow of the finger of the user produced by the second illumination; a touch point detection unit that, on the basis of the first shadow and the second shadow extracted, detects a touch point of the finger of the user on the operation surface; and a contour detection unit that detects a contour of the first shadow and a contour of the second shadow extracted, wherein the touch point detection unit detects a touch point of the finger of the user on the operation surface from the distance between the contour of the first shadow and the contour of the second shadow wherein the shadow region extraction unit compares the brightness of the captured image with a predetermined threshold value, and discerns and extracts the first shadow of the finger of the user and the second shadow of the finger of the user, wherein the contour detection unit extracts, as contours, corresponding substantially linear line segments from within the contour of the first shadow and the contour of the second shadow, and wherein the touch point detection unit determines that the finger of the user has touched the operation surface when the distance between the two contours extracted has become equal to or less than a predetermined threshold value.
 7. The projector according to claim 6, wherein the first illumination and the second illumination are made to irradiate in a temporally alternating manner, the camera captures the operation surface in accordance with irradiation timings of each of the irradiation of the first illumination and the irradiation of the second illumination, and the shadow region extraction unit extracts the first shadow from an image captured by the first illumination, and extracts the second shadow from an image captured by the second illumination temporally separated from the image captured by the first illumination.
 8. The projector according to claim 6, wherein the first illumination and the second illumination are installed in such a way that the illumination directions thereof are oriented toward substantially the same side as the imaging direction of the camera. 